Skip to content

Robotics: the cap-harness engine, the benchmarks and ASPIRE no longer share code - #125

Merged
haizhongzheng merged 6 commits into
mainfrom
robotics-separate
Nov 21, 2026
Merged

haizhongzheng merged 6 commits into
mainfrom
robotics-separate

Conversation

@haizhongzheng

@haizhongzheng haizhongzheng commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Change

The robot benchmarks and the ASPIRE method shared code. ASPIRE's fix loop, evolutionary search and verifier lived in the Robosuite benchmark's task functions, and the benchmark's prepare.py wrote ASPIRE's verifier rows. So another method on LIBERO had to strip ASPIRE's parts (evosearch: false, deleting task.md), and LIBERO ran on another example's code. Now a benchmark holds nothing of a method, and ASPIRE nothing of a benchmark.

Where What
examples/cap_harness/ (new) The engine Robosuite and LIBERO run on. It holds the simulator service, bin/sim, neutral task functions (prepare, collect), the judge, the task writer (utils/benchmark/tasks.py), the row view (fields.py), and launch.sh, which sources a method folder's tasks.sh. RoboDojo's service subclasses its service. Its README has the engine's setup.
examples/robosuite/ (was aspire_robosuite) The Robosuite tasks (prepare.py), task-info, and an aspire/ folder of configs.
examples/libero/ Its tasks, its neutral prompt, and an aspire/ and a prompt_only/ folder. The prompt-only configs use the engine's workspace and record as they are.
hermit/agent/evolver/methods/aspire/ actor.py holds ASPIRE's acting sessions (fix-loop and verifier prompts, the search, the facts the coordinator reads), built on the engine's task functions (agent.params.engine). rows.py derives ASPIRE's verifier rows before a run, through the engine's task writer. direct.py is the direct-chat comparison actor.

Interfaces:

  • A benchmark's prepare.py writes its tasks and their reruns (rerun-<id>, rerun_of). Reruns stay with the benchmark, because a test split may not share ids with the training split, and prompt-only uses them too.
  • ASPIRE's verifier rows (verify-<id>) come from rows.py, which aspire/tasks.sh runs into that folder's tasks/ and splits/.
  • The simulator service's attempt budget is a row field, max_programs (it used to follow the row's role). The judge scores a row's eval_seeds, so a verifier row sets them to its own five, with score_label: verifier for the feedback.
  • The verifier verdict rule moved from the judge to ASPIRE's collect and direct actor (not_scored): an unverified verifier's program is withheld.
  • Core: hermit/agent/acting/workspace.py accepts a task function named package.module:function, matched by a strict dotted regex, so ASPIRE's configs name hermit.agent.evolver.methods.aspire.actor:prompt. tests/selection.py gains ENGINES: a change to examples/cap_harness/ runs the groups of the methods on its benchmarks.
  • tests/core/test_environment.py reads the source of a package.module:Class agent without importing it.

Two behavioural notes:

  • No runtime text changes. ASPIRE's prompt pages, the coordinator's guide and the templates are byte-identical to main's (prompts/*.md compared file by file). A {context_file} line for Claude Code was reverted along with the dropped Claude config. Run A ran with that line, so its Claude sessions were told CLAUDE.md; at this head a Claude Code session would be told AGENTS.md.
  • No Claude Code config in this PR. A smoke_claude.yaml was dropped: it needs CLAUDE_CODE_OAUTH_TOKEN to run as committed (task sandbox, login: claude), and it could not be run here. Run A below used Claude Code with a run-only config.

Validation

Parity

  • Prompts are byte-identical to main. Checked 13 on the test checkout, and 8 on ASPIRE's real templates at the pinned commit fa00f0ef (216 KB): actor, no search, max_steps, LIBERO, verifier ×2, neutral ×2.
  • prepare() workspaces are identical in 5 cases (ASPIRE robo / no search / LIBERO / verifier, and prompt-only); only the per-run path and port in bin/sim.json differ. bin/sim_cli.py, which is copied into every workspace, is unchanged.
  • Row counts are as before: Robosuite 7 tasks, 7 reruns and 7 verifiers = 21; LIBERO 210 + 210 + 210 = 630.

Tests

  • python -m pytest --changed --base origin/main on ecc8cd94 (the review fixes and a merge of main). The two later commits only drop smoke_claude.yaml and revert the {context_file} line, and the affected tests pass (86, then 45). selected every group: 1797 passed, 148 skipped, 7 failed. All 7 fail the same way on a clean main worktree in this environment:
    • an older hermit is installed in site-packages: test_the_moved_modules_old_paths_are_gone ×4;
    • the upstream harness2 package is missing: test_harness2_on_pi ×2;
    • test_rrsi_sees_a_harbor_rollouts_failure... ×1.
  • After the last changes: tests/methods/aspire, examples/cap_harness/utils/tests, RoboDojo's tests, test_layers and test_environment: 93 passed.

Entry points

  • All three launchers (robosuite/aspire, libero/prompt_only, libero/aspire) run end to end with stub providers and a wrapper in place of the final hermit train: prepare, tasks.sh, the staged docs, the simulator service up and healthy, and the train command.
  • hermit check --no-probe passes on one config per folder against the real ASPIRE checkout.

GPU, no model

  • On an L40, the real simulator service ran: registration, then bin/sim on a development seed, a held-out seed refused, the unseen check once, the engine's collect, and the judge on held-out seeds 1-3 (a fixed minimal program scored 0/3, as expected).
  • A verifier row's third program was refused by max_programs.

Real smoke runs (Robosuite ASPIRE smoke: two tasks, ASPIRE's providers on free L40 GPUs)

Both run-only configs below were deleted after the runs; neither is committed.

Run Policy and coordinator Sandbox cube_lifting cube_stack
A Claude Opus 5.5 on this machine's own Claude Code sign-in (host runner) off 1.0: 3/3 held-out, unseen 1/1 1.0: 3/3 held-out, unseen 1/1
B Codex, GPT-5.6-Luna on the ChatGPT login on 1.0: 3/3 held-out, unseen 1/1 0.0: the fix loop fell short; the judge scored the evolutionary search's best program
  • Run A took 19 minutes. In it, the coordinator ran ASPIRE's verifier on each task (the verify-* rows that rows.py derived), the judge scored the verified programs, and it promoted entries into grasp.md, transport.md and localize.md.
  • Run B checks the task-sandbox path of ASPIRE's module-named task functions. Inside the sandbox, actor.prompt built the prompt, prepare built the workspace, and the actor ran bin/sim. The launch records report probe.passed: true for rollout-1 and learner-1/-2.

Found, not fixed (predates this PR; worth issues)

  • A CLI actor with environment.runner: host puts its workspace under .logs/workspaces, outside the directory the simulator service serves, so the service refuses it. Run A passed agent.params.workspaces=$ASPIRE_OUT/actor-logs/rollout.
  • ASPIRE's evolver.yaml sets home: [], so a coordinator without a login or binding gets no sign-in. Run A set evolver.params.home: [.credentials.json].
  • Codex 0.170, as installed here (~/.local/bin/codex is a symlink into its standalone package), needs codex-code-mode-host beside its real binary. Hermit's task sandbox shows only the binary, so a sandboxed Codex session has no tools. Run B set agent.params.bin to the real path and added its folder to environment.read.

Review

  • Tests and documentation cover the affected behavior:
    • the engine tests (25) and ASPIRE's (18); the old 38 are all kept, two of them split into an engine half and an ASPIRE half;
    • docs: CHANGELOG (Breaking), docs/methods/aspire.md, docs/framework/task.md, examples/README.md, the engine, Robosuite and LIBERO READMEs, tests/README.md.
  • Commands, defaults, and public interfaces are consistent: config paths moved, with no compatibility layer (as agreed for this refactor); ASPIRE_* launcher variables unchanged.
  • Agents follow the agent layers: tests/core/test_layers.py passes (the agent layer imports no task layer). The direct actor's own loop predates this PR (a comparison; cf. Move the benchmarks' own agent loops to ReactAgent subclasses (CL-Bench, ServeLearnBench, OfficeQA, AgentDyn) #122).
  • Shared logs and examples contain no credentials or private data.
  • Sandbox: no process added; the task sandbox and every cap-harness run (sim.confine) are as before. Run B's launch records report probe.passed: true (rollout and learner); run A ran unsandboxed, by the user's choice.
  • Benchmark: tasks are written as task directories (write_task); network and timeouts are unchanged; parity as above.
  • Independent review: /dual-review ran 3 rounds; each round was this session plus a fresh, read-only Claude Opus 5.5 session.
    • Round 1:
      • fixed: the docs said every session is Codex;
      • smoke_claude.yaml could not be run in its committed form (it needs CLAUDE_CODE_OAUTH_TOKEN), so it was dropped from this PR (5100ea94).
    • Round 2:
      • fixed: RoboDojo's README called its base "the ASPIRE example's service";
      • fixed: direct.py's docstring overstated its scope (it is Robosuite's);
      • Claude Code's Bash timeout (2 min by default, 10 at most) against long bin/sim run-batches: no timeouts in run A. A Claude Code config for full runs should raise it (BASH_DEFAULT_TIMEOUT_MS/BASH_MAX_TIMEOUT_MS in the session's env).
    • Round 3: no new finding.
    • Minor points left: a literal prompt shaped like a.b:c would be read as a module function; reruns' instruction.md wording; the record of an unverified verifier no longer carries fix_code (it stays in files/); rows.py's own seeds_of; the launcher does not check for a Claude token up front.
    • An independent audit before the PR found the earlier issues listed under Change (all fixed).

Not changed, for the maintainer's review (agents read this text at run time):

  • the docstring of bin/sim_cli.py still says "a verifier's two programs";
  • LIBERO's task-info mentions skill_report for every record;
  • Robosuite's task-info is titled "ASPIRE on Robosuite".

🤖 Generated with Claude Code

haizhongzheng and others added 6 commits November 21, 2026 01:56
… share code

A benchmark now holds nothing of a method, and ASPIRE nothing of a benchmark.

- examples/cap_harness/: the engine Robosuite and LIBERO run on (simulator
  service, bin/sim, neutral task functions, judge, task writer, row view, the
  launcher with a method folder's tasks.sh hook); RoboDojo's service builds on it.
- examples/robosuite/ (was aspire_robosuite): the Robosuite tasks and an aspire/
  folder. A benchmark's prepare.py writes its tasks and their reruns (rerun_of).
- hermit/agent/evolver/methods/aspire/: actor.py (ASPIRE's fix-loop, search and
  verifier sessions over the engine's task functions, agent.params.engine; the
  verdict rule withholds an unverified verifier's program), rows.py (ASPIRE's
  verifier rows, derived before a run), direct.py (the direct-chat comparison).
- The service's attempt budget is a row's max_programs; the judge scores a row's
  eval_seeds. A task function may be named package.module:function.
- The Codex sessions' prompts and workspaces are unchanged byte for byte;
  ASPIRE's header names the CLI's context file (CLAUDE.md for Claude Code);
  Robosuite gains a Claude Code smoke config.
- Tests: the engine's in examples/cap_harness/utils/tests (selected through
  ENGINES), ASPIRE's in tests/methods/aspire.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ase service, the direct actor's scope; Claude's Bash timeout

- The engine and Robosuite READMEs and actor.py said every session is Codex;
  they now name Claude Code too (smoke_claude.yaml).
- RoboDojo's README and two comments still called its base "the ASPIRE example's
  service"; it is the cap-harness engine's.
- direct.py's docstring said any cap-harness task; its prompts and runs are
  Robosuite's.
- smoke_claude.yaml raises Claude Code's Bash timeout (2 min by default, 10 at
  most) through the session's environment, since a bin/sim run-batch can take
  longer.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…tted

smoke_claude.yaml (task sandbox, login: claude) needs CLAUDE_CODE_OAUTH_TOKEN,
which the machine the PR was validated on does not have, so it never ran in the
form it was committed in. ASPIRE's header still names the CLI's context file
(CLAUDE.md for Claude Code); a Claude Code config can follow once it can run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The {context_file} line existed only for a Claude Code session, and this PR no
longer ships a Claude Code config. ASPIRE's prompt pages are now byte-identical
to main's; the Claude Code wording can come with a Claude Code config.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The index said every folder under examples/ is one task; cap_harness/ is the
engine Robosuite and LIBERO run on, and claude_subscription/ holds a guide's
configs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@haizhongzheng
haizhongzheng merged commit 20e060b into main Nov 21, 2026
1 check passed
@haizhongzheng
haizhongzheng deleted the robotics-separate branch November 21, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant